Min-Cut Max-Flow for Network Abnormality Detection: Application to Preterm Birth

Neuroimaging studies of structural connectomes typically average the data from many subjects and analyse the average properties of the resulting network. We propose a new framework for individual brain-network structural abnormality detection. The framework uses a graph-based anomaly detection algorithm that allows to detect abnormal structural connectivity on a subject level. The proposed method is generic and can be adapted for a broad range of network abnormality detection problems. In this study, we apply our method to investigate the integrity of white matter tracts of 19-year-old extremely preterm born individuals. We show the feasibility to cast the network abnormality detection problem into a min-cut max-flow problem, and identify consistent abnormal white matter tracts in extremely preterm subjects, including a common network involving the bilateral thalamus and frontal gyri

Image Compositing for Segmentation of Surgical Tools Without Manual Annotations

Producing manual, pixel-accurate, image segmentation labels is tedious and time-consuming. This is often a rate-limiting factor when large amounts of labeled images are required, such as for training deep convolutional networks for instrument-background segmentation in surgical scenes. No large datasets comparable to industry standards in the computer vision community are available for this task. To circumvent this problem, we propose to automate the creation of a realistic training dataset by exploiting techniques stemming from special effects and harnessing them to target training performance rather than visual appeal. Foreground data is captured by placing sample surgical instruments over a chroma key (a.k.a. green screen) in a controlled environment, thereby making extraction of the relevant image segment straightforward. Multiple lighting conditions and viewpoints can be captured and introduced in the simulation by moving the instruments and camera and modulating the light source. Background data is captured by collecting videos that do not contain instruments. In the absence of pre-existing instrument-free background videos, minimal labeling effort is required, just to select frames that do not contain surgical instruments from videos of surgical interventions freely available online. We compare different methods to blend instruments over tissue and propose a novel data augmentation approach that takes advantage of the plurality of options. We show that by training a vanilla U-Net on semi-synthetic data only and applying a simple post-processing, we are able to match the results of the same network trained on a publicly available manually labeled real dataset

A spatio-temporal atlas of the developing fetal brain with spina bifida aperta [version 2; peer review: 2 approved]

Background: Spina bifida aperta (SBA) is a birth defect associated with severe anatomical changes in the developing fetal brain. Brain magnetic resonance imaging (MRI) atlases are popular tools for studying neuropathology in the brain anatomy, but previous fetal brain MRI atlases have focused on the normal fetal brain. We aimed to develop a spatio-temporal fetal brain MRI atlas for SBA. Methods: We developed a semi-automatic computational method to compute the first spatio-temporal fetal brain MRI atlas for SBA. We used 90 MRIs of fetuses with SBA with gestational ages ranging from 21 to 35 weeks. Isotropic and motion-free 3D reconstructed MRIs were obtained for all the examinations. We propose a protocol for the annotation of anatomical landmarks in brain 3D MRI of fetuses with SBA with the aim of making spatial alignment of abnormal fetal brain MRIs more robust. In addition, we propose a weighted generalized Procrustes method based on the anatomical landmarks for the initialization of the atlas. The proposed weighted generalized Procrustes can handle temporal regularization and missing annotations. After initialization, the atlas is refined iteratively using non-linear image registration based on the image intensity and the anatomical land-marks. A semi-automatic method is used to obtain a parcellation of our fetal brain atlas into eight tissue types: white matter, ventricular system, cerebellum, extra-axial cerebrospinal fluid, cortical gray matter, deep gray matter, brainstem, and corpus callosum. Results: An intra-rater variability analysis suggests that the seven anatomical land-marks are sufficiently reliable. We find that the proposed atlas outperforms a normal fetal brain atlas for the automatic segmentation of brain 3D MRI of fetuses with SBA. Conclusions: We make publicly available a spatio-temporal fetal brain MRI atlas for SBA, available here: https://doi.org/10.7303/syn25887675. This atlas can support future research on automatic segmentation methods for brain 3D MRI of fetuses with SBA

A spatio-temporal atlas of the developing fetal brain with spina bifida aperta

Background: Spina bifida aperta (SBA) is a birth defect associated with severe anatomical changes in the developing fetal brain. Brain magnetic resonance imaging (MRI) atlases are popular tools for studying neuropathology in the brain anatomy, but previous fetal brain MRI atlases have focused on the normal fetal brain. We aimed to develop a spatio-temporal fetal brain MRI atlas for SBA. Methods: We developed a semi-automatic computational method to compute the first spatio-temporal fetal brain MRI atlas for SBA. We used 90 MRIs of fetuses with SBA with gestational ages ranging from 21 to 35 weeks. Isotropic and motion-free 3D reconstructed MRIs were obtained for all the examinations. We propose a protocol for the annotation of anatomical landmarks in brain 3D MRI of fetuses with SBA with the aim of making spatial alignment of abnormal fetal brain MRIs more robust. In addition, we propose a weighted generalized Procrustes method based on the anatomical landmarks for the initialization of the atlas. The proposed weighted generalized Procrustes can handle temporal regularization and missing annotations. After initialization, the atlas is refined iteratively using non-linear image registration based on the image intensity and the anatomical land-marks. A semi-automatic method is used to obtain a parcellation of our fetal brain atlas into eight tissue types: white matter, ventricular system, cerebellum, extra-axial cerebrospinal fluid, cortical gray matter, deep gray matter, brainstem, and corpus callosum. Results: An intra-rater variability analysis suggests that the seven anatomical land-marks are sufficiently reliable. We find that the proposed atlas outperforms a normal fetal brain atlas for the automatic segmentation of brain 3D MRI of fetuses with SBA. Conclusions: We make publicly available a spatio-temporal fetal brain MRI atlas for SBA, available here: https://doi.org/10.7303/syn25887675. This atlas can support future research on automatic segmentation methods for brain 3D MRI of fetuses with SBA

A spatio-temporal atlas of the developing fetal brain with spina bifida aperta

Background: Spina bifida aperta (SBA) is a birth defect associated with severe anatomical changes in the developing fetal brain. Brain magnetic resonance imaging (MRI) atlases are popular tools for studying neuropathology in the brain anatomy, but previous fetal brain MRI atlases have focused on the normal fetal brain. We aimed to develop a spatio-temporal fetal brain MRI atlas for SBA. Methods: We developed a semi-automatic computational method to compute the first spatio-temporal fetal brain MRI atlas for SBA. We used 90 MRIs of fetuses with SBA with gestational ages ranging from 21 to 35 weeks. Isotropic and motion-free 3D reconstructed MRIs were obtained for all the examinations. We propose a protocol for the annotation of anatomical landmarks in brain 3D MRI of fetuses with SBA with the aim of making spatial alignment of abnormal fetal brain MRIs more robust. In addition, we propose a weighted generalized Procrustes method based on the anatomical landmarks for the initialization of the atlas. The proposed weighted generalized Procrustes can handle temporal regularization and missing annotations. After initialization, the atlas is refined iteratively using non-linear image registration based on the image intensity and the anatomical land-marks. A semi-automatic method is used to obtain a parcellation of our fetal brain atlas into eight tissue types: white matter, ventricular system, cerebellum, extra-axial cerebrospinal fluid, cortical gray matter, deep gray matter, brainstem, and corpus callosum. Results: An intra-rater variability analysis suggests that the seven anatomical land-marks are sufficiently reliable. We find that the proposed atlas outperforms a normal fetal brain atlas for the automatic segmentation of brain 3D MRI of fetuses with SBA. Conclusions: We make publicly available a spatio-temporal fetal brain MRI atlas for SBA, available here: https://doi.org/10.7303/syn25887675. This atlas can support future research on automatic segmentation methods for brain 3D MRI of fetuses with SBA

Cortical Surface Matching of the Fetal Brain Pre and Post Fetal Surgery for Open Spina Bifida

Introduction: Fetal surgery has become a clinical reality, even for non-lethal conditions such as open spina bifidayelomeningocele, the spinal cord extrudes into a cereberospinal fluid (CSF) filled sac 1, 2. It is associated with brain anomalies such as hindbrain herniation and variable degrees of ventriculomegaly. Prenatalrepair yields better outcomes compared to postnatal surgery3. Nonetheless, mechanical tissue damage of brain parenchyma due to ventriculomegaly and damage to the neural tracts lead to abnormal white matter development, as demonstrated by diffusion weighted imaging studies4-9. This may lead to altered gyrification patterns in MMC patients. Gyrification, measured by magnetic resonance imaging (MRI), correlates with motor and cognitive function in infants, children and adolescents with who have undergone postnatal closure 6. Evaluation of cognitiveand motor function in fetuses who have had prenatal surgery, is performed only after birth, with deficits becoming more evident with increasing age. Clinicians therefore urgently need early fetal brain imaging methods that can predict the cognitive and motor challenges that fetuses may encounter after birth. We aim to demonstrate that longitudinal quantitative MRI measurement of cortical gyrification is possible, before and after repair. We will also demonstrate the curvature (curvedness and shape index) of cerebellum and ventricles before and after surgery. Methods: T2-weighted single-shot fast spin-echo (SSFSE) was performed of the fetal brain in multiple containing an axial, coronal and sagittal planewith 3mm slice thicknesson women with , both before surgery (n=12, 23+6 1+7 weeks, (22+1–25+6)) and after surgery (n=12, 26+1 1+3 weeks, (24+1– 29+4))acquisition time thirty-forty minutes. Initial diagnosis of open spina bifida was made on mid-trimester ultrasound. Fetuses affected by aneuploidy or with structural anomalies outside the CNS were excluded. A novel automated super resolution reconstruction (SRR) algorithm10, 11 was used to build 3D volumes of the fetal brain based on the 2D stacks that were acquired in different directions. Rigid slice-to-volume registration correcting for fetal motion was used to generate an SRR image in standard anatomical orientation, from which we automatically segmented white matter, ventricles, and cerebellum using template brain segmentations12, 13. 14make us of 12Brain masks for pre-operative SRR volumes were resampled from their corresponding post-operative MMC masks after affine and non-rigid alignment. All masks where manually corrected and meshes were generated using ITK-Snap14. A rigid coherent point drift algorithm was applied to find an initial correspondence for the intrasubject cortical, cerebellar and ventricle regions before and after surgery. Joint spectral matching (JSM) was then used to find the correspondence for the intrasubject at those two different time points. In JSM a dual layered graph was produced whereby layers correspond to the surface of the white matter, cerebellum or ventricles of each subject. The correspondence links from the initial intrasubject matching, connecting both layers to produce a set of shared eigenmodes of the surfaces. After mapping the post-operative surface to the pre-operative surface using JSM, we computed the change in parameters at the vertex of each mesh to explore longitudinal cortical gyrification, and curvature (curvedness and shape index) of the 12. Results: Figure 1 illustrates five spectral modes for the white matter, ventricles and cerebellum of a fetus before surgery (24 weeks), and therafter (26 weeks). Although the meshes are quite different in the three-dimensional space, with respect to different levels of folding, variation in shape, surface area and volume, they have similar representations in the spectral domain. Figure 2 shows the curvedness and shape index in the white matter, cerebellum and ventricles before and after fetal surgery. JSM allows us to map the mean curvatures of each mesh to compute changes in the mean and to generate the shape index.Figure 3 shows maps of mean curvature of a fetus pre and post-surgery. Positive values are depicted in red/yellow and represent gyri (convex structures), and negative values in blue represent sulci (concave) structures. JSM allows mapping of mean curvatures from the post-op to the pre-op space, computing the changes in mean curvature between these two time points in the pre-op space.Figure 4 illustrates the shape index histogram for white matter, showing the differences in gyri and sulci between the pre and post-operative MMC brain.Discussion: Surface-based matching provides additional information about changes in growth and gyrification of the fetal brain compared to measurement of total volume and shape change. This may be useful in evaluating changes in cerebral growth of MMC fetuses before and after fetal surgery. Spectral graph matching is a promising tool for matching shapes with significant differences in cortical folding, surface area, and volume, but with similar representations in the spectral domain such as depicted with fetuses before and after surgery12, 15. Future work may be able to better explore the physiological and mechanical properties contributing to the differences observed in brain growth and development in the context of fetal surgery. Conclusion: Novel analysis of fetal longitudinal correspondence of white matter, and development of specific regions of the brain as secondary gyri emerges, in the context of fetal surgery is demonstrated. This tool allows the measurement of the shape and growth of the white matter surface may help establish longitudinal growth trajectories. References: 1. Rethmann, C., et al., Evolution of posterior fossa and brain morphology after in utero repair of open neural tube defects assessed by MRI. Eur Radiol, 2017. 27(11): p. 4571-4580. 2. Zarutskie, A., et al., Prenatal brain imaging for predicting need for postnatal hydrocephalus treatment in fetuses that had neural tube defect repair in utero. Ultrasound Obstet Gynecol, 2019. 53(3): p. 324-334. 3. Adzick, N.S.T., E.A.; Spong, C. Y.; Brock III, J. W.; Burrows, P. K.; Johnson, M. P.; Howell, R. N.; Farrell, J. N.; Dabrowiak, M.E.; Sutton, L.N.; Gupta, N.; Tulipan, N.B.; D’Alton, M.E.; Farmer, D.L., A Randomised Trial of Prenatal versus Postnatal Repair of Myelomeningocele. N Engl J Med, 2011. 364: p. 993-1004. 4. Juranek, J., et al., Neocortical reorganization in spina bifida. Neuroimage, 2008. 40(4): p. 1516-22. 5. Juranek, J. and M.S. Salman, Anomalous development of brain structure and function in spina bifida myelomeningocele. Developmental Disabilities Research Reviews, 2010. 16(1): p. 23-30. 6. Treble, A., et al., Functional significance of atypical cortical organization in spina bifida myelomeningocele: relations of cortical thickness and gyrification with IQ and fine motor dexterity. Cereb Cortex, 2013. 23(10): p. 2357-69. 7. Hasan, K.M., et al., White matter microstructural abnormalities in children with spina bifida myelomeningocele and hydrocephalus: a diffusion tensor tractography study of the association pathways. J Magn Reson Imaging, 2008. 27(4): p. 700-9. 8. Mignone Philpott, C., et al., Diffusion-weighted imaging of the cerebellum in the fetus with Chiari II malformation. AJNR Am J Neuroradiol, 2013. 34(8): p. 1656-60. 9. Woitek, R., et al., Fetal diffusion tensor quantification of brainstem pathology in Chiari II malformation. Eur Radiol, 2016. 26(5): p. 1274-83. 10. Ebner, M., et al., An Automated Localization, Segmentation and Reconstruction Framework for Fetal Brain MRI, in Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. 2018. p. 313-320. 11. Ebner, M., et al., An automated framework for localization, segmentation and super-resolution reconstruction of fetal brain MRI. NeuroImage, 2019. 12. Orasanu, E., et al., Cortical folding of the preterm brain: a longitudinal analysis of extremely preterm born neonates using spectral matching. Brain Behav, 2016. 6(8): p. e00488. 13. Kuklisova-Murgasova, M., et al., A dynamic 4D probabilistic atlas of the developing brain. Neuroimage, 2011. 54(4): p. 2750-63. 14. Yushkevich, P.A., et al., User-guided 3D active contour segmentation of anatomical structures: significantly improved efficiency and reliability. Neuroimage, 2006. 31(3): p. 1116-28. 15. Lomabert, H., Spopring J., and Siddiqi K., Diffeomorphic spectral matching of cortical surafaces. Inf Process Med Imaging, 2013. 7917: p. 376-289

Label-Set Loss Functions for Partial Supervision: Application to Fetal Brain 3D MRI Parcellation

Deep neural networks have increased the accuracy of automatic segmentation, however their accuracy depends on the availability of a large number of fully segmented images. Methods to train deep neural networks using images for which some, but not all, regions of interest are segmented are necessary to make better use of partially annotated datasets. In this paper, we propose the first axiomatic definition of label-set loss functions that are the loss functions that can handle partially segmented images. We prove that there is one and only one method to convert a classical loss function for fully segmented images into a proper label-set loss function. Our theory also allows us to define the leaf-Dice loss, a label-set generalisation of the Dice loss particularly suited for partial supervision with only missing labels. Using the leaf-Dice loss, we set a new state of the art in partially supervised learning for fetal brain 3D MRI segmentation. We achieve a deep neural network able to segment white matter, ventricles, cerebellum, extra-ventricular CSF, cortical gray matter, deep gray matter, brainstem, and corpus callosum based on fetal brain 3D MRI of anatomically normal fetuses or with open spina bifida. Our implementation of the proposed label-set loss functions is available at https://github.com/LucasFidon/label-set-loss-functions

Label-Set Loss Functions for Partial Supervision: Application to Fetal Brain 3D MRI Parcellation

Deep neural networks have increased the accuracy of automatic segmentation, however their accuracy depends on the availability of a large number of fully segmented images. Methods to train deep neural networks using images for which some, but not all, regions of interest are segmented are necessary to make better use of partially annotated datasets. In this paper, we propose the first axiomatic definition of label-set loss functions that are the loss functions that can handle partially segmented images. We prove that there is one and only one method to convert a classical loss function for fully segmented images into a proper label-set loss function. Our theory also allows us to define the leaf-Dice loss, a label-set generalisation of the Dice loss particularly suited for partial supervision with only missing labels. Using the leaf-Dice loss, we set a new state of the art in partially supervised learning for fetal brain 3D MRI segmentation. We achieve a deep neural network able to segment white matter, ventricles, cerebellum, extra-ventricular CSF, cortical gray matter, deep gray matter, brainstem, and corpus callosum based on fetal brain 3D MRI of anatomically normal fetuses or with open spina bifida. Our implementation of the proposed label-set loss functions is available at https://github.com/LucasFidon/label-set-loss-functions

Distributionally Robust Deep Learning using Hardness Weighted Sampling

Limiting failures of machine learning systems is vital for safety-critical applications. In order to improve the robustness of machine learning systems, Distributionally Robust Optimization (DRO) has been proposed as a generalization of Empirical Risk Minimization (ERM)aiming at addressing this need. However, its use in deep learning has been severely restricted due to the relative inefficiency of the optimizers available for DRO in comparison to the wide-spread variants of Stochastic Gradient Descent (SGD) optimizers for ERM. We propose SGD with hardness weighted sampling, a principled and efficient optimization method for DRO in machine learning that is particularly suited in the context of deep learning. Similar to a hard example mining strategy in essence and in practice, the proposed algorithm is straightforward to implement and computationally as efficient as SGD-based optimizers used for deep learning, requiring minimal overhead computation. In contrast to typical ad hoc hard mining approaches, and exploiting recent theoretical results in deep learning optimization, we prove the convergence of our DRO algorithm for over-parameterized deep learning networks with ReLU activation and finite number of layers and parameters. Our experiments on brain tumor segmentation in MRI demonstrate the feasibility and the usefulness of our approach. Using our hardness weighted sampling leads to a decrease of 2% of the interquartile range of the Dice scores for the enhanced tumor and the tumor core regions. The code for the proposed hard weighted sampler will be made publicly available

A Dempster-Shafer approach to trustworthy AI with application to fetal brain MRI segmentation

Deep learning models for medical image segmentation can fail unexpectedly and spectacularly for pathological cases and images acquired at different centers than training images, with labeling errors that violate expert knowledge. Such errors undermine the trustworthiness of deep learning models for medical image segmentation. Mechanisms for detecting and correcting such failures are essential for safely translating this technology into clinics and are likely to be a requirement of future regulations on artificial intelligence (AI). In this work, we propose a trustworthy AI theoretical framework and a practical system that can augment any backbone AI system using a fallback method and a fail-safe mechanism based on Dempster-Shafer theory. Our approach relies on an actionable definition of trustworthy AI. Our method automatically discards the voxel-level labeling predicted by the backbone AI that violate expert knowledge and relies on a fallback for those voxels. We demonstrate the effectiveness of the proposed trustworthy AI approach on the largest reported annotated dataset of fetal MRI consisting of 540 manually annotated fetal brain 3D T2w MRIs from 13 centers. Our trustworthy AI method improves the robustness of a state-of-the-art backbone AI for fetal brain MRIs acquired across various centers and for fetuses with various brain abnormalities